# -.-|m { output: false, input_fold: hide }
%load_ext pretty_jupyter
#Imports
%matplotlib inline
import tensorflow as tf
from tensorflow.keras.applications.resnet_v2 import ResNet50V2, preprocess_input
from tensorflow.keras.preprocessing import image_dataset_from_directory
from tensorflow.keras.layers import GlobalAveragePooling2D, Dense, Dropout
from tensorflow.data.experimental import cardinality
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import RandomFlip, RandomRotation
from tensorflow.keras import Input, Model
# from tensorflow.keras.optimizers import Adam, RMSprop
from tensorflow.keras.optimizers.legacy import Adam, RMSprop
from tensorflow.keras.losses import BinaryCrossentropy
from tensorflow.keras.metrics import BinaryAccuracy
import matplotlib.pyplot as plt
from sklearn.metrics import classification_report, ConfusionMatrixDisplay
import numpy as np
Introduction¶
In this assignment, we will look into the detection of melanoma using a Convolutional Neural Network (CNN) by applying transfer learning with a pre-trained model. I decided using the 🔬Melanoma Cancer Image Dataset. It consists of 13 900 high-resolution images in test and train sets, both divided in two categorical fodlers - Benign and Malignant. The test folders comprise ot 1 000 photos each. The images are uniformly sized at 224 x 224 pixels.
The development of a CNN (Convolutional Neural Network) model for melanoma detection is motivated by the pressing need for accurate and efficient tools in the field of dermatology and healthcare. Melanoma, a type of skin cancer, poses significant health risks if not detected and treated early. However, manual diagnosis of melanoma by dermatologists can be time-consuming and prone to errors, leading to delays in treatment and potentially adverse outcomes for patients.
CNN models offer a promising solution by leveraging machine learning techniques to automatically analyze skin lesion images and identify potential signs of melanoma. These models can be trained on large datasets of annotated images, learning to recognize patterns and features indicative of melanoma with high accuracy.
For the choice of the pre-trained model I searched for previous uses and researches. Based on the performance comparison in the Sagar (2020) research I decided to opt for the ResNet50V2 model.
Pre-process¶
Defining variables¶
TRAIN_PATH = 'data/train'
VAL_PATH = 'data/validation'
TEST_PATH = 'data/test'
IMAGE_SIZE = (224, 224)
BATCH_SIZE = 32
CHANNELS = 3
EPOCHS = 10
Reading the datasets¶
train_ds = image_dataset_from_directory(
TRAIN_PATH,
image_size = IMAGE_SIZE,
batch_size = BATCH_SIZE
)
val_ds = image_dataset_from_directory(
VAL_PATH,
image_size = IMAGE_SIZE,
batch_size = BATCH_SIZE
)
test_ds = image_dataset_from_directory(
TEST_PATH,
image_size = IMAGE_SIZE,
batch_size = BATCH_SIZE
)
Found 11180 files belonging to 2 classes. Found 1600 files belonging to 2 classes. Found 400 files belonging to 2 classes.
class_names = train_ds.class_names
print(class_names)
['Benign', 'Malignant']
for data_batch, labels_batch in train_ds:
print("data batch shape:", data_batch.shape)
print("labels batch shape:", labels_batch.shape)
break
data batch shape: (32, 224, 224, 3) labels batch shape: (32,)
labels_batch
<tf.Tensor: shape=(32,), dtype=int32, numpy=
array([1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1,
1, 0, 0, 0, 0, 0, 1, 0, 1, 1], dtype=int32)>
train_ds
<_PrefetchDataset element_spec=(TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32, name=None), TensorSpec(shape=(None,), dtype=tf.int32, name=None))>
Plotting images¶
plt.figure(figsize=(10, 10))
for images, labels in train_ds.take(1):
for i in range(9):
ax = plt.subplot(3, 3, i + 1)
plt.imshow(images[i].numpy().astype("uint8"))
plt.title(class_names[labels[i]])
plt.axis("off")
Data augmentation¶
We will apply some minor image transformation in order to expose the model to varying data examples in order to reduce overfitting. For this we use the RandomFlip and RandomRotation layers.
data_augmentation = Sequential([
RandomFlip('horizontal'),
RandomRotation(0.2),
])
Image preprocessing¶
The ResNet50V2 model expects pixel values to be between -1 and 1. For this we will use the models preprocess_input function to prepare everything for us.
preprocess_input = preprocess_input
Model¶
Creating the base model¶
Loading the ResNet50V2 model with the our datasets image size, without the classification layers at the top (which is ideal for feature extraction) and with the weights from the imagenet dataset training.
IMAGE_SHAPE = IMAGE_SIZE + (3,)
base_model = ResNet50V2(input_shape=IMAGE_SHAPE,
include_top=False,
weights='imagenet')
We need to freeze the convolutional base before compiling and training the model to use it as a feature extractor afterwards. Since the model has many layers we set the trainable flag to False for the whole modle.
base_model.trainable = False
base_model.summary()
Model: "resnet50v2"
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) [(None, 224, 224, 3)] 0 []
conv1_pad (ZeroPadding2D) (None, 230, 230, 3) 0 ['input_1[0][0]']
conv1_conv (Conv2D) (None, 112, 112, 64) 9472 ['conv1_pad[0][0]']
pool1_pad (ZeroPadding2D) (None, 114, 114, 64) 0 ['conv1_conv[0][0]']
pool1_pool (MaxPooling2D) (None, 56, 56, 64) 0 ['pool1_pad[0][0]']
conv2_block1_preact_bn (Ba (None, 56, 56, 64) 256 ['pool1_pool[0][0]']
tchNormalization)
conv2_block1_preact_relu ( (None, 56, 56, 64) 0 ['conv2_block1_preact_bn[0][0]
Activation) ']
conv2_block1_1_conv (Conv2 (None, 56, 56, 64) 4096 ['conv2_block1_preact_relu[0][
D) 0]']
conv2_block1_1_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block1_1_conv[0][0]']
rmalization)
conv2_block1_1_relu (Activ (None, 56, 56, 64) 0 ['conv2_block1_1_bn[0][0]']
ation)
conv2_block1_2_pad (ZeroPa (None, 58, 58, 64) 0 ['conv2_block1_1_relu[0][0]']
dding2D)
conv2_block1_2_conv (Conv2 (None, 56, 56, 64) 36864 ['conv2_block1_2_pad[0][0]']
D)
conv2_block1_2_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block1_2_conv[0][0]']
rmalization)
conv2_block1_2_relu (Activ (None, 56, 56, 64) 0 ['conv2_block1_2_bn[0][0]']
ation)
conv2_block1_0_conv (Conv2 (None, 56, 56, 256) 16640 ['conv2_block1_preact_relu[0][
D) 0]']
conv2_block1_3_conv (Conv2 (None, 56, 56, 256) 16640 ['conv2_block1_2_relu[0][0]']
D)
conv2_block1_out (Add) (None, 56, 56, 256) 0 ['conv2_block1_0_conv[0][0]',
'conv2_block1_3_conv[0][0]']
conv2_block2_preact_bn (Ba (None, 56, 56, 256) 1024 ['conv2_block1_out[0][0]']
tchNormalization)
conv2_block2_preact_relu ( (None, 56, 56, 256) 0 ['conv2_block2_preact_bn[0][0]
Activation) ']
conv2_block2_1_conv (Conv2 (None, 56, 56, 64) 16384 ['conv2_block2_preact_relu[0][
D) 0]']
conv2_block2_1_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block2_1_conv[0][0]']
rmalization)
conv2_block2_1_relu (Activ (None, 56, 56, 64) 0 ['conv2_block2_1_bn[0][0]']
ation)
conv2_block2_2_pad (ZeroPa (None, 58, 58, 64) 0 ['conv2_block2_1_relu[0][0]']
dding2D)
conv2_block2_2_conv (Conv2 (None, 56, 56, 64) 36864 ['conv2_block2_2_pad[0][0]']
D)
conv2_block2_2_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block2_2_conv[0][0]']
rmalization)
conv2_block2_2_relu (Activ (None, 56, 56, 64) 0 ['conv2_block2_2_bn[0][0]']
ation)
conv2_block2_3_conv (Conv2 (None, 56, 56, 256) 16640 ['conv2_block2_2_relu[0][0]']
D)
conv2_block2_out (Add) (None, 56, 56, 256) 0 ['conv2_block1_out[0][0]',
'conv2_block2_3_conv[0][0]']
conv2_block3_preact_bn (Ba (None, 56, 56, 256) 1024 ['conv2_block2_out[0][0]']
tchNormalization)
conv2_block3_preact_relu ( (None, 56, 56, 256) 0 ['conv2_block3_preact_bn[0][0]
Activation) ']
conv2_block3_1_conv (Conv2 (None, 56, 56, 64) 16384 ['conv2_block3_preact_relu[0][
D) 0]']
conv2_block3_1_bn (BatchNo (None, 56, 56, 64) 256 ['conv2_block3_1_conv[0][0]']
rmalization)
conv2_block3_1_relu (Activ (None, 56, 56, 64) 0 ['conv2_block3_1_bn[0][0]']
ation)
conv2_block3_2_pad (ZeroPa (None, 58, 58, 64) 0 ['conv2_block3_1_relu[0][0]']
dding2D)
conv2_block3_2_conv (Conv2 (None, 28, 28, 64) 36864 ['conv2_block3_2_pad[0][0]']
D)
conv2_block3_2_bn (BatchNo (None, 28, 28, 64) 256 ['conv2_block3_2_conv[0][0]']
rmalization)
conv2_block3_2_relu (Activ (None, 28, 28, 64) 0 ['conv2_block3_2_bn[0][0]']
ation)
max_pooling2d (MaxPooling2 (None, 28, 28, 256) 0 ['conv2_block2_out[0][0]']
D)
conv2_block3_3_conv (Conv2 (None, 28, 28, 256) 16640 ['conv2_block3_2_relu[0][0]']
D)
conv2_block3_out (Add) (None, 28, 28, 256) 0 ['max_pooling2d[0][0]',
'conv2_block3_3_conv[0][0]']
conv3_block1_preact_bn (Ba (None, 28, 28, 256) 1024 ['conv2_block3_out[0][0]']
tchNormalization)
conv3_block1_preact_relu ( (None, 28, 28, 256) 0 ['conv3_block1_preact_bn[0][0]
Activation) ']
conv3_block1_1_conv (Conv2 (None, 28, 28, 128) 32768 ['conv3_block1_preact_relu[0][
D) 0]']
conv3_block1_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block1_1_conv[0][0]']
rmalization)
conv3_block1_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block1_1_bn[0][0]']
ation)
conv3_block1_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block1_1_relu[0][0]']
dding2D)
conv3_block1_2_conv (Conv2 (None, 28, 28, 128) 147456 ['conv3_block1_2_pad[0][0]']
D)
conv3_block1_2_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block1_2_conv[0][0]']
rmalization)
conv3_block1_2_relu (Activ (None, 28, 28, 128) 0 ['conv3_block1_2_bn[0][0]']
ation)
conv3_block1_0_conv (Conv2 (None, 28, 28, 512) 131584 ['conv3_block1_preact_relu[0][
D) 0]']
conv3_block1_3_conv (Conv2 (None, 28, 28, 512) 66048 ['conv3_block1_2_relu[0][0]']
D)
conv3_block1_out (Add) (None, 28, 28, 512) 0 ['conv3_block1_0_conv[0][0]',
'conv3_block1_3_conv[0][0]']
conv3_block2_preact_bn (Ba (None, 28, 28, 512) 2048 ['conv3_block1_out[0][0]']
tchNormalization)
conv3_block2_preact_relu ( (None, 28, 28, 512) 0 ['conv3_block2_preact_bn[0][0]
Activation) ']
conv3_block2_1_conv (Conv2 (None, 28, 28, 128) 65536 ['conv3_block2_preact_relu[0][
D) 0]']
conv3_block2_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block2_1_conv[0][0]']
rmalization)
conv3_block2_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block2_1_bn[0][0]']
ation)
conv3_block2_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block2_1_relu[0][0]']
dding2D)
conv3_block2_2_conv (Conv2 (None, 28, 28, 128) 147456 ['conv3_block2_2_pad[0][0]']
D)
conv3_block2_2_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block2_2_conv[0][0]']
rmalization)
conv3_block2_2_relu (Activ (None, 28, 28, 128) 0 ['conv3_block2_2_bn[0][0]']
ation)
conv3_block2_3_conv (Conv2 (None, 28, 28, 512) 66048 ['conv3_block2_2_relu[0][0]']
D)
conv3_block2_out (Add) (None, 28, 28, 512) 0 ['conv3_block1_out[0][0]',
'conv3_block2_3_conv[0][0]']
conv3_block3_preact_bn (Ba (None, 28, 28, 512) 2048 ['conv3_block2_out[0][0]']
tchNormalization)
conv3_block3_preact_relu ( (None, 28, 28, 512) 0 ['conv3_block3_preact_bn[0][0]
Activation) ']
conv3_block3_1_conv (Conv2 (None, 28, 28, 128) 65536 ['conv3_block3_preact_relu[0][
D) 0]']
conv3_block3_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block3_1_conv[0][0]']
rmalization)
conv3_block3_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block3_1_bn[0][0]']
ation)
conv3_block3_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block3_1_relu[0][0]']
dding2D)
conv3_block3_2_conv (Conv2 (None, 28, 28, 128) 147456 ['conv3_block3_2_pad[0][0]']
D)
conv3_block3_2_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block3_2_conv[0][0]']
rmalization)
conv3_block3_2_relu (Activ (None, 28, 28, 128) 0 ['conv3_block3_2_bn[0][0]']
ation)
conv3_block3_3_conv (Conv2 (None, 28, 28, 512) 66048 ['conv3_block3_2_relu[0][0]']
D)
conv3_block3_out (Add) (None, 28, 28, 512) 0 ['conv3_block2_out[0][0]',
'conv3_block3_3_conv[0][0]']
conv3_block4_preact_bn (Ba (None, 28, 28, 512) 2048 ['conv3_block3_out[0][0]']
tchNormalization)
conv3_block4_preact_relu ( (None, 28, 28, 512) 0 ['conv3_block4_preact_bn[0][0]
Activation) ']
conv3_block4_1_conv (Conv2 (None, 28, 28, 128) 65536 ['conv3_block4_preact_relu[0][
D) 0]']
conv3_block4_1_bn (BatchNo (None, 28, 28, 128) 512 ['conv3_block4_1_conv[0][0]']
rmalization)
conv3_block4_1_relu (Activ (None, 28, 28, 128) 0 ['conv3_block4_1_bn[0][0]']
ation)
conv3_block4_2_pad (ZeroPa (None, 30, 30, 128) 0 ['conv3_block4_1_relu[0][0]']
dding2D)
conv3_block4_2_conv (Conv2 (None, 14, 14, 128) 147456 ['conv3_block4_2_pad[0][0]']
D)
conv3_block4_2_bn (BatchNo (None, 14, 14, 128) 512 ['conv3_block4_2_conv[0][0]']
rmalization)
conv3_block4_2_relu (Activ (None, 14, 14, 128) 0 ['conv3_block4_2_bn[0][0]']
ation)
max_pooling2d_1 (MaxPoolin (None, 14, 14, 512) 0 ['conv3_block3_out[0][0]']
g2D)
conv3_block4_3_conv (Conv2 (None, 14, 14, 512) 66048 ['conv3_block4_2_relu[0][0]']
D)
conv3_block4_out (Add) (None, 14, 14, 512) 0 ['max_pooling2d_1[0][0]',
'conv3_block4_3_conv[0][0]']
conv4_block1_preact_bn (Ba (None, 14, 14, 512) 2048 ['conv3_block4_out[0][0]']
tchNormalization)
conv4_block1_preact_relu ( (None, 14, 14, 512) 0 ['conv4_block1_preact_bn[0][0]
Activation) ']
conv4_block1_1_conv (Conv2 (None, 14, 14, 256) 131072 ['conv4_block1_preact_relu[0][
D) 0]']
conv4_block1_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block1_1_conv[0][0]']
rmalization)
conv4_block1_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block1_1_bn[0][0]']
ation)
conv4_block1_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block1_1_relu[0][0]']
dding2D)
conv4_block1_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block1_2_pad[0][0]']
D)
conv4_block1_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block1_2_conv[0][0]']
rmalization)
conv4_block1_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block1_2_bn[0][0]']
ation)
conv4_block1_0_conv (Conv2 (None, 14, 14, 1024) 525312 ['conv4_block1_preact_relu[0][
D) 0]']
conv4_block1_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block1_2_relu[0][0]']
D)
conv4_block1_out (Add) (None, 14, 14, 1024) 0 ['conv4_block1_0_conv[0][0]',
'conv4_block1_3_conv[0][0]']
conv4_block2_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block1_out[0][0]']
tchNormalization)
conv4_block2_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block2_preact_bn[0][0]
Activation) ']
conv4_block2_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block2_preact_relu[0][
D) 0]']
conv4_block2_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block2_1_conv[0][0]']
rmalization)
conv4_block2_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block2_1_bn[0][0]']
ation)
conv4_block2_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block2_1_relu[0][0]']
dding2D)
conv4_block2_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block2_2_pad[0][0]']
D)
conv4_block2_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block2_2_conv[0][0]']
rmalization)
conv4_block2_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block2_2_bn[0][0]']
ation)
conv4_block2_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block2_2_relu[0][0]']
D)
conv4_block2_out (Add) (None, 14, 14, 1024) 0 ['conv4_block1_out[0][0]',
'conv4_block2_3_conv[0][0]']
conv4_block3_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block2_out[0][0]']
tchNormalization)
conv4_block3_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block3_preact_bn[0][0]
Activation) ']
conv4_block3_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block3_preact_relu[0][
D) 0]']
conv4_block3_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block3_1_conv[0][0]']
rmalization)
conv4_block3_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block3_1_bn[0][0]']
ation)
conv4_block3_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block3_1_relu[0][0]']
dding2D)
conv4_block3_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block3_2_pad[0][0]']
D)
conv4_block3_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block3_2_conv[0][0]']
rmalization)
conv4_block3_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block3_2_bn[0][0]']
ation)
conv4_block3_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block3_2_relu[0][0]']
D)
conv4_block3_out (Add) (None, 14, 14, 1024) 0 ['conv4_block2_out[0][0]',
'conv4_block3_3_conv[0][0]']
conv4_block4_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block3_out[0][0]']
tchNormalization)
conv4_block4_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block4_preact_bn[0][0]
Activation) ']
conv4_block4_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block4_preact_relu[0][
D) 0]']
conv4_block4_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block4_1_conv[0][0]']
rmalization)
conv4_block4_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block4_1_bn[0][0]']
ation)
conv4_block4_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block4_1_relu[0][0]']
dding2D)
conv4_block4_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block4_2_pad[0][0]']
D)
conv4_block4_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block4_2_conv[0][0]']
rmalization)
conv4_block4_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block4_2_bn[0][0]']
ation)
conv4_block4_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block4_2_relu[0][0]']
D)
conv4_block4_out (Add) (None, 14, 14, 1024) 0 ['conv4_block3_out[0][0]',
'conv4_block4_3_conv[0][0]']
conv4_block5_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block4_out[0][0]']
tchNormalization)
conv4_block5_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block5_preact_bn[0][0]
Activation) ']
conv4_block5_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block5_preact_relu[0][
D) 0]']
conv4_block5_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block5_1_conv[0][0]']
rmalization)
conv4_block5_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block5_1_bn[0][0]']
ation)
conv4_block5_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block5_1_relu[0][0]']
dding2D)
conv4_block5_2_conv (Conv2 (None, 14, 14, 256) 589824 ['conv4_block5_2_pad[0][0]']
D)
conv4_block5_2_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block5_2_conv[0][0]']
rmalization)
conv4_block5_2_relu (Activ (None, 14, 14, 256) 0 ['conv4_block5_2_bn[0][0]']
ation)
conv4_block5_3_conv (Conv2 (None, 14, 14, 1024) 263168 ['conv4_block5_2_relu[0][0]']
D)
conv4_block5_out (Add) (None, 14, 14, 1024) 0 ['conv4_block4_out[0][0]',
'conv4_block5_3_conv[0][0]']
conv4_block6_preact_bn (Ba (None, 14, 14, 1024) 4096 ['conv4_block5_out[0][0]']
tchNormalization)
conv4_block6_preact_relu ( (None, 14, 14, 1024) 0 ['conv4_block6_preact_bn[0][0]
Activation) ']
conv4_block6_1_conv (Conv2 (None, 14, 14, 256) 262144 ['conv4_block6_preact_relu[0][
D) 0]']
conv4_block6_1_bn (BatchNo (None, 14, 14, 256) 1024 ['conv4_block6_1_conv[0][0]']
rmalization)
conv4_block6_1_relu (Activ (None, 14, 14, 256) 0 ['conv4_block6_1_bn[0][0]']
ation)
conv4_block6_2_pad (ZeroPa (None, 16, 16, 256) 0 ['conv4_block6_1_relu[0][0]']
dding2D)
conv4_block6_2_conv (Conv2 (None, 7, 7, 256) 589824 ['conv4_block6_2_pad[0][0]']
D)
conv4_block6_2_bn (BatchNo (None, 7, 7, 256) 1024 ['conv4_block6_2_conv[0][0]']
rmalization)
conv4_block6_2_relu (Activ (None, 7, 7, 256) 0 ['conv4_block6_2_bn[0][0]']
ation)
max_pooling2d_2 (MaxPoolin (None, 7, 7, 1024) 0 ['conv4_block5_out[0][0]']
g2D)
conv4_block6_3_conv (Conv2 (None, 7, 7, 1024) 263168 ['conv4_block6_2_relu[0][0]']
D)
conv4_block6_out (Add) (None, 7, 7, 1024) 0 ['max_pooling2d_2[0][0]',
'conv4_block6_3_conv[0][0]']
conv5_block1_preact_bn (Ba (None, 7, 7, 1024) 4096 ['conv4_block6_out[0][0]']
tchNormalization)
conv5_block1_preact_relu ( (None, 7, 7, 1024) 0 ['conv5_block1_preact_bn[0][0]
Activation) ']
conv5_block1_1_conv (Conv2 (None, 7, 7, 512) 524288 ['conv5_block1_preact_relu[0][
D) 0]']
conv5_block1_1_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block1_1_conv[0][0]']
rmalization)
conv5_block1_1_relu (Activ (None, 7, 7, 512) 0 ['conv5_block1_1_bn[0][0]']
ation)
conv5_block1_2_pad (ZeroPa (None, 9, 9, 512) 0 ['conv5_block1_1_relu[0][0]']
dding2D)
conv5_block1_2_conv (Conv2 (None, 7, 7, 512) 2359296 ['conv5_block1_2_pad[0][0]']
D)
conv5_block1_2_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block1_2_conv[0][0]']
rmalization)
conv5_block1_2_relu (Activ (None, 7, 7, 512) 0 ['conv5_block1_2_bn[0][0]']
ation)
conv5_block1_0_conv (Conv2 (None, 7, 7, 2048) 2099200 ['conv5_block1_preact_relu[0][
D) 0]']
conv5_block1_3_conv (Conv2 (None, 7, 7, 2048) 1050624 ['conv5_block1_2_relu[0][0]']
D)
conv5_block1_out (Add) (None, 7, 7, 2048) 0 ['conv5_block1_0_conv[0][0]',
'conv5_block1_3_conv[0][0]']
conv5_block2_preact_bn (Ba (None, 7, 7, 2048) 8192 ['conv5_block1_out[0][0]']
tchNormalization)
conv5_block2_preact_relu ( (None, 7, 7, 2048) 0 ['conv5_block2_preact_bn[0][0]
Activation) ']
conv5_block2_1_conv (Conv2 (None, 7, 7, 512) 1048576 ['conv5_block2_preact_relu[0][
D) 0]']
conv5_block2_1_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block2_1_conv[0][0]']
rmalization)
conv5_block2_1_relu (Activ (None, 7, 7, 512) 0 ['conv5_block2_1_bn[0][0]']
ation)
conv5_block2_2_pad (ZeroPa (None, 9, 9, 512) 0 ['conv5_block2_1_relu[0][0]']
dding2D)
conv5_block2_2_conv (Conv2 (None, 7, 7, 512) 2359296 ['conv5_block2_2_pad[0][0]']
D)
conv5_block2_2_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block2_2_conv[0][0]']
rmalization)
conv5_block2_2_relu (Activ (None, 7, 7, 512) 0 ['conv5_block2_2_bn[0][0]']
ation)
conv5_block2_3_conv (Conv2 (None, 7, 7, 2048) 1050624 ['conv5_block2_2_relu[0][0]']
D)
conv5_block2_out (Add) (None, 7, 7, 2048) 0 ['conv5_block1_out[0][0]',
'conv5_block2_3_conv[0][0]']
conv5_block3_preact_bn (Ba (None, 7, 7, 2048) 8192 ['conv5_block2_out[0][0]']
tchNormalization)
conv5_block3_preact_relu ( (None, 7, 7, 2048) 0 ['conv5_block3_preact_bn[0][0]
Activation) ']
conv5_block3_1_conv (Conv2 (None, 7, 7, 512) 1048576 ['conv5_block3_preact_relu[0][
D) 0]']
conv5_block3_1_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block3_1_conv[0][0]']
rmalization)
conv5_block3_1_relu (Activ (None, 7, 7, 512) 0 ['conv5_block3_1_bn[0][0]']
ation)
conv5_block3_2_pad (ZeroPa (None, 9, 9, 512) 0 ['conv5_block3_1_relu[0][0]']
dding2D)
conv5_block3_2_conv (Conv2 (None, 7, 7, 512) 2359296 ['conv5_block3_2_pad[0][0]']
D)
conv5_block3_2_bn (BatchNo (None, 7, 7, 512) 2048 ['conv5_block3_2_conv[0][0]']
rmalization)
conv5_block3_2_relu (Activ (None, 7, 7, 512) 0 ['conv5_block3_2_bn[0][0]']
ation)
conv5_block3_3_conv (Conv2 (None, 7, 7, 2048) 1050624 ['conv5_block3_2_relu[0][0]']
D)
conv5_block3_out (Add) (None, 7, 7, 2048) 0 ['conv5_block2_out[0][0]',
'conv5_block3_3_conv[0][0]']
post_bn (BatchNormalizatio (None, 7, 7, 2048) 8192 ['conv5_block3_out[0][0]']
n)
post_relu (Activation) (None, 7, 7, 2048) 0 ['post_bn[0][0]']
==================================================================================================
Total params: 23564800 (89.89 MB)
Trainable params: 0 (0.00 Byte)
Non-trainable params: 23564800 (89.89 MB)
__________________________________________________________________________________________________
This featuer extractor turns every 224x224x3 input image into a 7x7x2048 block of features.
image_batch, label_batch = next(iter(train_ds))
feature_batch = base_model(image_batch)
print(feature_batch.shape)
(32, 7, 7, 2048)
Classification head¶
To generate predictions from the block of features we need to average over the spatial 7x7 spatial locations, using a GlobalAveragePooling2D layer to convert the features to a single 2048-element vector per image.
global_average_layer = GlobalAveragePooling2D()
feature_batch_average = global_average_layer(feature_batch)
print(feature_batch_average.shape)
(32, 2048)
Subsequently, we employ a Dense layer to transform these features into a singular prediction for each image. We only have two classes so we will use the sigmoid function and will use the prediction as a raw prediction value. Positive values indicate a prediction for class 1, while negative values indicate a prediction for class 0.
prediction_layer = Dense(1, activation='sigmoid')
prediction_batch = prediction_layer(feature_batch_average)
print(prediction_batch.shape)
(32, 1)
Now we need to build a model that incorporates all the steps: data augmentation, preprocessing, base_model and feature extractor.
Because the model contains BatchNormalization layers, we need to set training=False for the base model so that the updates applied to the non-trainable weights do not destroy what the model has learned.
inputs = Input(shape=(224, 224, 3))
x = data_augmentation(inputs)
x = preprocess_input(x)
x = base_model(x, training=False)
x = global_average_layer(x)
x = Dropout(0.2)(x)
outputs = prediction_layer(x)
model = Model(inputs, outputs)
Now to see what the model looks like:
model.summary()
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, 224, 224, 3)] 0
sequential (Sequential) (None, 224, 224, 3) 0
tf.math.truediv (TFOpLambd (None, 224, 224, 3) 0
a)
tf.math.subtract (TFOpLamb (None, 224, 224, 3) 0
da)
resnet50v2 (Functional) (None, 7, 7, 2048) 23564800
global_average_pooling2d ( (None, 2048) 0
GlobalAveragePooling2D)
dropout (Dropout) (None, 2048) 0
dense (Dense) (None, 1) 2049
=================================================================
Total params: 23566849 (89.90 MB)
Trainable params: 2049 (8.00 KB)
Non-trainable params: 23564800 (89.89 MB)
_________________________________________________________________
We can see that the ResNet50V2 parameters are frozen and we only have the 2 049 trainable ones from our Dense layer that are the weights and biases.
Compiling the model¶
We need to compile before the training and since we have only two classes and a sigmoid raw value => we use BinaryAccuracy to determine the result. We use the Adam optimizer and a learning_rate or 0.0001.
Using the legacy Adam optimizer as the new one runs slower on the M1 chip.
base_learning_rate = 0.0001
model.compile(optimizer=Adam(learning_rate=base_learning_rate),
loss=BinaryCrossentropy(),
metrics=[BinaryAccuracy(threshold=0.5, name='accuracy')])
We start by training the model over 10 epochs.
# -.-|m { output: false }
history = model.fit(train_ds,
validation_data=val_ds,
epochs=EPOCHS)
Learning curves¶
Now we can look at the learning curves to see the training and validation accuracy/loss.
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
plt.figure(figsize=(8, 8))
plt.subplot(2, 1, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.ylabel('Accuracy')
plt.ylim([min(plt.ylim()),1])
plt.title('Training and Validation Accuracy')
plt.subplot(2, 1, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.ylabel('Cross Entropy')
plt.ylim([0,1.0])
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.show()
We can see that the validation metrics are better than the training ones but we can say that this is because the BatchNormalization and Dropout layers affect accuracy during training and are turned off when calculating the validation loss.
Fine tuning the model¶
In order to increase the performacne we can fite tune the model. Till here we have only trained the layers that we added without updating any weights from the base_model.
Lets look at the number of layers in the base_model and only select a small amount of the higher up layers as those are the more specialized ones (whereas the first ones are more simple and catch more generic features).
For that we will unfreeze the model, check the number of layers in the base and select for which ones to disable the training again.
base_model.trainable = True
print("Number of layers in the base model: ", len(base_model.layers))
Number of layers in the base model: 190
fine_tune_from = 170
for layer in base_model.layers[:fine_tune_from]:
layer.trainable = False
Now we need to compile the new model and check the amount of trainable parameters. As we are training a larger model and adapting pretrained weights, we need to lower the learning rate so that we do not overfit the model.
We will once again use the legacy optimizer as the new one is slow on the M1.
model.compile(loss=BinaryCrossentropy(),
optimizer = RMSprop(learning_rate=base_learning_rate/10),
metrics=[BinaryAccuracy(threshold=0.5, name='accuracy')])
model.summary()
Model: "model"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
input_2 (InputLayer) [(None, 224, 224, 3)] 0
sequential (Sequential) (None, 224, 224, 3) 0
tf.math.truediv (TFOpLambd (None, 224, 224, 3) 0
a)
tf.math.subtract (TFOpLamb (None, 224, 224, 3) 0
da)
resnet50v2 (Functional) (None, 7, 7, 2048) 23564800
global_average_pooling2d ( (None, 2048) 0
GlobalAveragePooling2D)
dropout (Dropout) (None, 2048) 0
dense (Dense) (None, 1) 2049
=================================================================
Total params: 23566849 (89.90 MB)
Trainable params: 7881729 (30.07 MB)
Non-trainable params: 15685120 (59.83 MB)
_________________________________________________________________
Now we add 10 additional epochs and continue the training from where we had left off.
# -.-|m { output: false }
fine_tune_epochs = 10
total_epochs = EPOCHS + fine_tune_epochs
history_fine = model.fit(train_ds,
epochs=total_epochs,
initial_epoch=history.epoch[-1],
validation_data=val_ds)
Learning curve comparisson¶
Now we can look at the learning curves once again for the training and validation and compare them to the ones from before the fine tuning.
acc += history_fine.history['accuracy']
val_acc += history_fine.history['val_accuracy']
loss += history_fine.history['loss']
val_loss += history_fine.history['val_loss']
plt.figure(figsize=(8, 8))
plt.subplot(2, 1, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.ylim([0.8, 1])
plt.plot([EPOCHS-1,EPOCHS-1],
plt.ylim(), label='Start Fine Tuning')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')
plt.subplot(2, 1, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.ylim([0, 1.0])
plt.plot([EPOCHS-1,EPOCHS-1],
plt.ylim(), label='Start Fine Tuning')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.show()
We can see the validation accuracy drop towards the 19th epoch whereas the training one gets higher, so we might be starting to overfit the model. That may be due to the fact that our training set is quite small compared to the imagenet on which the base_model was trained.
model.save('ResNet50V2-Melanoma-FineTune-CNN.keras')
Evaluation and prediction¶
We can check the performance of the model with our test_ds to check how it performs with new data.
loss, accuracy = model.evaluate(test_ds)
print('Test accuracy :', accuracy)
13/13 [==============================] - 4s 296ms/step - loss: 0.2482 - accuracy: 0.9000 Test accuracy : 0.8999999761581421
And now let's see it make some predictions in action. We make sure to have all the predictions, labels and images in one cell to avoid the shuffling of images and labels.
y_test = np.array([])
pred_test = np.array([])
all_images = []
for x , y in test_ds:
y_test = np.concatenate([y_test, y])
pred_test = np.concatenate([pred_test, model.predict(x, verbose=0).reshape(-1,)])
all_images.append(x)
test_images = np.concatenate(all_images)
test_images.shape
(400, 224, 224, 3)
plt.hist(pred_test)
plt.show()
As we are doing binary classification we only need two classes for the results:
predictions = np.where(pred_test >0.5,1,0).reshape(-1,)
print(classification_report(y_test, predictions))
precision recall f1-score support
0.0 0.89 0.92 0.90 200
1.0 0.91 0.89 0.90 200
accuracy 0.90 400
macro avg 0.90 0.90 0.90 400
weighted avg 0.90 0.90 0.90 400
Confusion matrix¶
To better compare the predictions and labels we will use a confusion matrix.
ConfusionMatrixDisplay.from_predictions(y_test, predictions)
plt.show()
Let's also look at some of the misclassified images:
misclass = y_test !=predictions
misclassified_images = [i for i, x in enumerate(misclass) if x==True]
misclassified_images[:10]
[12, 14, 31, 42, 48, 60, 68, 69, 72, 82]
plt.figure(figsize=(10,10))
for i , number in enumerate(misclassified_images[:9]):
ax = plt.subplot(3,3,i+1)
plt.imshow(test_images[number].astype("uint8"))
plt.axis("off")
plt.show()
Conclusion¶
From the confusion matrix we can compare predictions to labels and see how many are wrong. From the results we can see that the model has a 90% accuracy and gave false predictions for only 40 out of the 400 test samples.
The models performance was somewhat good when freezing all layers of the base_model but further improved with the fine tuning until it started overfitting towards the end.
We might get better results if we retrain a larger amount of higher base_model layers.
References¶
🔬Melanoma Cancer Image Dataset. (2024, February 18). Kaggle. https://www.kaggle.com/datasets/bhaveshmittal/melanoma-cancer-dataset
Leo, M. S. (2022, March 19). How to choose the best Keras Pre-Trained Model for image Classification. Medium. https://towardsdatascience.com/how-to-choose-the-best-keras-pre-trained-model-for-image-classification-b850ca4428d4
Brownlee, J. (2019b, September 16). A gentle introduction to transfer learning for deep learning. MachineLearningMastery.com. https://machinelearningmastery.com/transfer-learning-for-deep-learning/
Brownlee, J. (2020, August 18). Transfer Learning in Keras with Computer Vision Models. MachineLearningMastery.com. https://machinelearningmastery.com/how-to-use-transfer-learning-when-developing-convolutional-neural-network-models/
Load and preprocess images. (n.d.). TensorFlow. https://www.tensorflow.org/tutorials/load_data/images
Module: tf.keras.applications.resnet_v2. (n.d.). TensorFlow. https://www.tensorflow.org/api_docs/python/tf/keras/applications/resnet_v2
Sagar, A. (2020). Convolutional neural networks for classifying melanoma images. bioRxiv (Cold Spring Harbor Laboratory). https://doi.org/10.1101/2020.05.22.110973
Transfer learning and fine-tuning. (n.d.). TensorFlow. https://www.tensorflow.org/tutorials/images/transfer_learning